viruPVS/langchain-docs huggingface dataset:
langchain_community.document_loaders.baiducloud_bos_file.BaiduBOSFileLoader¶
class langchain_community.document_loaders.baiducloud_bos_file.BaiduBOSFileLoader(conf: Any, bucket: str, key: str)[source]¶
Load from Baidu Cloud BOS file.
Initialize with BOS config, bucket and key name.
:param conf(BceClientConfiguration): BOS config.
:param bucket(str): BOS bucket.
:param key(str): BOS file key.
Methods
__init__(conf, bucket, key)
Initialize with BOS config, bucket and key name.
lazy_load()
Load documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(conf: Any, bucket: str, key: str)[source]¶
Initialize with BOS config, bucket and key name.
:param conf(BceClientConfiguration): BOS config.
:param bucket(str): BOS bucket.
:param key(str): BOS file key.
lazy_load() → Iterator[Document][source]¶
Load documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.pdf.AmazonTextractPDFLoader¶
class langchain_community.document_loaders.pdf.AmazonTextractPDFLoader(file_path: str, textract_features: Optional[Sequence[str]] = None, client: Optional[Any] = None, credentials_profile_name: Optional[str] = None, region_name: Optional[str] = None, endpoint_url: Optional[str] = None, headers: Optional[Dict] = None)[source]¶
Load PDF files from a local file system, HTTP or S3.
To authenticate, the AWS client uses the following methods to
automatically load credentials:
https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
If a specific credential profile should be used, you must pass
the name of the profile from the ~/.aws/credentials file that is to be used.
Make sure the credentials / roles used have the required policies to
access the Amazon Textract service.
Example
Initialize the loader.
Parameters
file_path – A file, url or s3 path for input file
textract_features – Features to be used for extraction, each feature
should be passed as a str that conforms to the enum
Textract_Features, see amazon-textract-caller pkg
client – boto3 textract client (Optional)
credentials_profile_name – AWS profile name, if not default (Optional)
region_name – AWS region, eg us-east-1 (Optional)
endpoint_url – endpoint url for the textract service (Optional)
Attributes
source
Methods
__init__(file_path[, textract_features, ...])
Initialize the loader.
lazy_load()
Lazy load documents
load()
Load given path as pages.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, textract_features: Optional[Sequence[str]] = None, client: Optional[Any] = None, credentials_profile_name: Optional[str] = None, region_name: Optional[str] = None, endpoint_url: Optional[str] = None, headers: Optional[Dict] = None) → None[source]¶
Initialize the loader.
Parameters
file_path – A file, url or s3 path for input file
textract_features – Features to be used for extraction, each feature
should be passed as a str that conforms to the enum
Textract_Features, see amazon-textract-caller pkg
client – boto3 textract client (Optional)
credentials_profile_name – AWS profile name, if not default (Optional)
region_name – AWS region, eg us-east-1 (Optional)
endpoint_url – endpoint url for the textract service (Optional)
lazy_load() → Iterator[Document][source]¶
Lazy load documents
load() → List[Document][source]¶
Load given path as pages.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AmazonTextractPDFLoader¶
Amazon Textract
langchain_community.document_loaders.roam.RoamLoader¶
class langchain_community.document_loaders.roam.RoamLoader(path: str)[source]¶
Load Roam files from a directory.
Initialize with a path.
Methods
__init__(path)
Initialize with a path.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: str)[source]¶
Initialize with a path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using RoamLoader¶
Roam
langchain_community.document_loaders.parsers.language.javascript.JavaScriptSegmenter¶
class langchain_community.document_loaders.parsers.language.javascript.JavaScriptSegmenter(code: str)[source]¶
Code segmenter for JavaScript.
Methods
__init__(code)
extract_functions_classes()
is_valid()
simplify_code()
__init__(code: str)[source]¶
extract_functions_classes() → List[str][source]¶
is_valid() → bool[source]¶
simplify_code() → str[source]¶
langchain_community.document_loaders.parsers.pdf.PDFPlumberParser¶
class langchain_community.document_loaders.parsers.pdf.PDFPlumberParser(text_kwargs: Optional[Mapping[str, Any]] = None, dedupe: bool = False, extract_images: bool = False)[source]¶
Parse PDF with PDFPlumber.
Initialize the parser.
Parameters
text_kwargs – Keyword arguments to pass to pdfplumber.Page.extract_text()
dedupe – Avoiding the error of duplicate characters if dedupe=True.
Methods
__init__([text_kwargs, dedupe, extract_images])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(text_kwargs: Optional[Mapping[str, Any]] = None, dedupe: bool = False, extract_images: bool = False) → None[source]¶
Initialize the parser.
Parameters
text_kwargs – Keyword arguments to pass to pdfplumber.Page.extract_text()
dedupe – Avoiding the error of duplicate characters if dedupe=True.
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob – Blob instance
Returns
List of documents
langchain_community.document_loaders.baiducloud_bos_directory.BaiduBOSDirectoryLoader¶
class langchain_community.document_loaders.baiducloud_bos_directory.BaiduBOSDirectoryLoader(conf: Any, bucket: str, prefix: str = '')[source]¶
Load from Baidu BOS directory.
Initialize with BOS config, bucket and prefix.
:param conf(BosConfig): BOS config.
:param bucket(str): BOS bucket.
:param prefix(str): prefix.
Methods
__init__(conf, bucket[, prefix])
Initialize with BOS config, bucket and prefix.
lazy_load()
Load documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(conf: Any, bucket: str, prefix: str = '')[source]¶
Initialize with BOS config, bucket and prefix.
:param conf(BosConfig): BOS config.
:param bucket(str): BOS bucket.
:param prefix(str): prefix.
lazy_load() → Iterator[Document][source]¶
Load documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.pdf.UnstructuredPDFLoader¶
class langchain_community.document_loaders.pdf.UnstructuredPDFLoader(file_path: Union[str, List[str]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load PDF files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain_community.document_loaders import UnstructuredPDFLoader
loader = UnstructuredPDFLoader(“example.pdf”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-pdf
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, List[str]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
langchain_community.document_loaders.airbyte.AirbyteStripeLoader¶
class langchain_community.document_loaders.airbyte.AirbyteStripeLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Stripe using an Airbyte source connector.
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load data into Document objects.
load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AirbyteStripeLoader¶
Airbyte Question Answering
Airbyte Stripe
langchain_community.document_loaders.unstructured.UnstructuredAPIFileLoader¶
class langchain_community.document_loaders.unstructured.UnstructuredAPIFileLoader(file_path: Union[str, List[str]] = '', mode: str = 'single', url: str = 'https://api.unstructured.io/general/v0/general', api_key: str = '', **unstructured_kwargs: Any)[source]¶
Load files using Unstructured API.
By default, the loader makes a call to the hosted Unstructured API.
If you are running the unstructured API locally, you can change the
API rule by passing in the url parameter when you initialize the loader.
The hosted Unstructured API requires an API key. See
https://www.unstructured.io/api-key/ if you need to generate a key.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
```python
from langchain_community.document_loaders import UnstructuredAPIFileLoader
loader = UnstructuredFileAPILoader(“example.pdf”, mode=”elements”, strategy=”fast”, api_key=”MY_API_KEY”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition
https://www.unstructured.io/api-key/
https://github.com/Unstructured-IO/unstructured-api
Initialize with file path.
Methods
__init__([file_path, mode, url, api_key])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, List[str]] = '', mode: str = 'single', url: str = 'https://api.unstructured.io/general/v0/general', api_key: str = '', **unstructured_kwargs: Any)[source]¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using UnstructuredAPIFileLoader¶
Unstructured File
langchain_community.document_loaders.blackboard.BlackboardLoader¶
class langchain_community.document_loaders.blackboard.BlackboardLoader(blackboard_course_url: str, bbrouter: str, load_all_recursively: bool = True, basic_auth: Optional[Tuple[str, str]] = None, cookies: Optional[dict] = None, continue_on_failure: bool = False)[source]¶
Load a Blackboard course.
This loader is not compatible with all Blackboard courses. It is only
compatible with courses that use the new Blackboard interface.
To use this loader, you must have the BbRouter cookie. You can get this
cookie by logging into the course and then copying the value of the
BbRouter cookie from the browser’s developer tools.
Example
from langchain_community.document_loaders import BlackboardLoader
loader = BlackboardLoader(
    blackboard_course_url="https://blackboard.example.com/webapps/blackboard/execute/announcement?method=search&context=course_entry&course_id=_123456_1",
    bbrouter="expires:12345...",
)
documents = loader.load()
Initialize with blackboard course url.
The BbRouter cookie is required for most blackboard courses.
Parameters
blackboard_course_url – Blackboard course url.
bbrouter – BbRouter cookie.
load_all_recursively – If True, load all documents recursively.
basic_auth – Basic auth credentials.
cookies – Cookies.
continue_on_failure – whether to continue loading the sitemap if an error
occurs loading a url, emitting a warning instead of raising an
exception. Setting this to True makes the loader more robust, but also
may result in missing data. Default: False
Raises
ValueError – If blackboard course url is invalid.
Attributes
web_path
Methods
ValueError – If blackboard course url is invalid.
Attributes
web_path
Methods
__init__(blackboard_course_url, bbrouter[, ...])
Initialize with blackboard course url.
aload()
Load text from the urls in web_path async into Documents.
check_bs4()
Check if BeautifulSoup4 is installed.
download(path)
Download a file from an url.
fetch_all(urls)
Fetch all urls concurrently with rate limiting.
lazy_load()
Lazy load text from the url(s) in web_path.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_filename(url)
Parse the filename from an url.
scrape([parser])
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])
Fetch all urls, then return soups for all results.
__init__(blackboard_course_url: str, bbrouter: str, load_all_recursively: bool = True, basic_auth: Optional[Tuple[str, str]] = None, cookies: Optional[dict] = None, continue_on_failure: bool = False)[source]¶
Initialize with blackboard course url.
The BbRouter cookie is required for most blackboard courses.
Parameters
blackboard_course_url – Blackboard course url.
bbrouter – BbRouter cookie.
load_all_recursively – If True, load all documents recursively.
basic_auth – Basic auth credentials.
cookies – Cookies.
continue_on_failure – whether to continue loading the sitemap if an error
occurs loading a url, emitting a warning instead of raising an
exception. Setting this to True makes the loader more robust, but also
may result in missing data. Default: False
Raises
ValueError – If blackboard course url is invalid.
Raises
ValueError – If blackboard course url is invalid.
aload() → List[Document]¶
Load text from the urls in web_path async into Documents.
check_bs4() → None[source]¶
Check if BeautifulSoup4 is installed.
Raises
ImportError – If BeautifulSoup4 is not installed.
download(path: str) → None[source]¶
Download a file from an url.
Parameters
path – Path to the file.
async fetch_all(urls: List[str]) → Any¶
Fetch all urls concurrently with rate limiting.
lazy_load() → Iterator[Document]¶
Lazy load text from the url(s) in web_path.
load() → List[Document][source]¶
Load data into Document objects.
Returns
List of Documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
parse_filename(url: str) → str[source]¶
Parse the filename from an url.
Parameters
url – Url to parse the filename from.
Returns
The filename.
scrape(parser: Optional[str] = None) → Any¶
Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]¶
Fetch all urls, then return soups for all results.
Examples using BlackboardLoader¶
Blackboard
langchain_community.document_loaders.pdf.PDFPlumberLoader¶
class langchain_community.document_loaders.pdf.PDFPlumberLoader(file_path: str, text_kwargs: Optional[Mapping[str, Any]] = None, dedupe: bool = False, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Load PDF files using pdfplumber.
Initialize with a file path.
Attributes
source
Methods
__init__(file_path[, text_kwargs, dedupe, ...])
Initialize with a file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, text_kwargs: Optional[Mapping[str, Any]] = None, dedupe: bool = False, headers: Optional[Dict] = None, extract_images: bool = False) → None[source]¶
Initialize with a file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.pdf.BasePDFLoader¶
class langchain_community.document_loaders.pdf.BasePDFLoader(file_path: str, *, headers: Optional[Dict] = None)[source]¶
Base Loader class for PDF files.
If the file is a web path, it will download it to a temporary file, use it, thenclean up the temporary file after completion.
Initialize with a file path.
Parameters
file_path – Either a local, S3 or web path to a PDF file.
headers – Headers to use for GET request to download a file from a web path.
Attributes
source
Methods
__init__(file_path, *[, headers])
Initialize with a file path.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, *, headers: Optional[Dict] = None)[source]¶
Initialize with a file path.
Parameters
file_path – Either a local, S3 or web path to a PDF file.
headers – Headers to use for GET request to download a file from a web path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
abstract load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.pdf.PyPDFDirectoryLoader¶
class langchain_community.document_loaders.pdf.PyPDFDirectoryLoader(path: str, glob: str = '**/[!.]*.pdf', silent_errors: bool = False, load_hidden: bool = False, recursive: bool = False, extract_images: bool = False)[source]¶
Load a directory with PDF files using pypdf and chunks at character level.
Loader also stores page numbers in metadata.
Methods
__init__(path[, glob, silent_errors, ...])
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: str, glob: str = '**/[!.]*.pdf', silent_errors: bool = False, load_hidden: bool = False, recursive: bool = False, extract_images: bool = False)[source]¶
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
langchain_community.document_loaders.notebook.concatenate_cells¶
langchain_community.document_loaders.notebook.concatenate_cells(cell: dict, include_outputs: bool, max_output_length: int, traceback: bool) → str[source]¶
Combine cells information in a readable format ready to be used.
Parameters
cell – A dictionary
include_outputs – Whether to include the outputs of the cell.
max_output_length – Maximum length of the output to be displayed.
traceback – Whether to return a traceback of the error.
Returns
A string with the cell information.
langchain_community.document_loaders.azure_ai_data.AzureAIDataLoader¶
class langchain_community.document_loaders.azure_ai_data.AzureAIDataLoader(url: str, glob: Optional[str] = None)[source]¶
Load from Azure AI Data.
Initialize with URL to a data asset or storage location
.
Attributes
url
URL to the data asset or storage location.
glob_pattern
Optional glob pattern to select files.
Methods
__init__(url[, glob])
Initialize with URL to a data asset or storage location
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(url: str, glob: Optional[str] = None)[source]¶
Initialize with URL to a data asset or storage location
.
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.