viruPVS/langchain-docs huggingface dataset:
langchain_community.document_transformers.openai_functions.create_metadata_tagger¶
langchain_community.document_transformers.openai_functions.create_metadata_tagger(metadata_schema: Union[Dict[str, Any], Type[BaseModel]], llm: BaseLanguageModel, prompt: Optional[ChatPromptTemplate] = None, *, tagging_chain_kwargs: Optional[Dict] = None) → OpenAIMetadataTagger[source]¶
Create a DocumentTransformer that uses an OpenAI function chain to automatically
tag documents with metadata based on their content and an input schema.
Args:
metadata_schema: Either a dictionary or pydantic.BaseModel class. If a dictionaryis passed in, it’s assumed to already be a valid JsonSchema.
For best results, pydantic.BaseModels should have docstrings describing what
the schema represents and descriptions for the parameters.
llm: Language model to use, assumed to support the OpenAI function-calling API.Defaults to use “gpt-3.5-turbo-0613”
prompt: BasePromptTemplate to pass to the model.
Returns:An LLMChain that will pass the given function to the model.
Example:from langchain_community.chat_models import ChatOpenAI
from langchain_community.document_transformers import create_metadata_tagger
from langchain_core.documents import Document
schema = {
    "properties": {
        "movie_title": { "type": "string" },
        "critic": { "type": "string" },
        "tone": {
            "type": "string",
            "enum": ["positive", "negative"]
        },
        "rating": {
            "type": "integer",
            "description": "The number of stars the critic rated the movie"
        }
    },
    "required": ["movie_title", "critic", "tone"]
}
"required": ["movie_title", "critic", "tone"]
}
# Must be an OpenAI model that supports functions
llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
document_transformer = create_metadata_tagger(schema, llm)
original_documents = [
    Document(page_content="Review of The Bee Movie
By Roger Ebert
This is the greatest movie ever made. 4 out of 5 stars.”),Document(page_content=”Review of The Godfather
By Anonymous
This movie was super boring. 1 out of 5 stars.”, metadata={“reliable”: False}),]
enhanced_documents = document_transformer.transform_documents(original_documents)
Examples using create_metadata_tagger¶
OpenAI Functions Metadata Tagger
langchain_community.document_transformers.openai_functions.OpenAIMetadataTagger¶
class langchain_community.document_transformers.openai_functions.OpenAIMetadataTagger[source]¶
Bases: BaseDocumentTransformer, BaseModel
Extract metadata tags from document contents using OpenAI functions.
Example:from langchain_community.chat_models import ChatOpenAI
from langchain_community.document_transformers import OpenAIMetadataTagger
from langchain_core.documents import Document
schema = {
    "properties": {
        "movie_title": { "type": "string" },
        "critic": { "type": "string" },
        "tone": {
            "type": "string",
            "enum": ["positive", "negative"]
        },
        "rating": {
            "type": "integer",
            "description": "The number of stars the critic rated the movie"
        }
    },
    "required": ["movie_title", "critic", "tone"]
}
# Must be an OpenAI model that supports functions
llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
tagging_chain = create_tagging_chain(schema, llm)
document_transformer = OpenAIMetadataTagger(tagging_chain=tagging_chain)
original_documents = [
    Document(page_content="Review of The Bee Movie
By Roger Ebert
This is the greatest movie ever made. 4 out of 5 stars.”),Document(page_content=”Review of The Godfather
By Anonymous
This movie was super boring. 1 out of 5 stars.”, metadata={“reliable”: False}),]
enhanced_documents = document_transformer.transform_documents(original_documents)
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param tagging_chain: Any = None¶
The chain used to extract metadata from each document.
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep – set to True to make a deep copy of the model
Returns
new model instance
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Automatically extract and populate metadata
for each document according to the provided schema.
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
langchain_community.document_transformers.doctran_text_qa.DoctranQATransformer¶
class langchain_community.document_transformers.doctran_text_qa.DoctranQATransformer(openai_api_key: Optional[str] = None, openai_api_model: Optional[str] = None)[source]¶
Extract QA from text documents using doctran.
Parameters
openai_api_key – OpenAI API key. Can also be specified via environment variable
OPENAI_API_KEY.
Example
from langchain_community.document_transformers import DoctranQATransformer
# Pass in openai_api_key or set env var OPENAI_API_KEY
qa_transformer = DoctranQATransformer()
transformed_document = await qa_transformer.atransform_documents(documents)
Methods
__init__([openai_api_key, openai_api_model])
atransform_documents(documents, **kwargs)
Extracts QA from text documents using doctran.
transform_documents(documents, **kwargs)
Transform a list of documents.
__init__(openai_api_key: Optional[str] = None, openai_api_model: Optional[str] = None) → None[source]¶
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Extracts QA from text documents using doctran.
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
Examples using DoctranQATransformer¶
Doctran Interrogate Documents
langchain_community.document_transformers.beautiful_soup_transformer.BeautifulSoupTransformer¶
class langchain_community.document_transformers.beautiful_soup_transformer.BeautifulSoupTransformer[source]¶
Transform HTML content by extracting specific tags and removing unwanted ones.
Example
from langchain_community.document_transformers import BeautifulSoupTransformer
bs4_transformer = BeautifulSoupTransformer()
docs_transformed = bs4_transformer.transform_documents(docs)
Initialize the transformer.
This checks if the BeautifulSoup4 package is installed.
If not, it raises an ImportError.
Methods
__init__()
Initialize the transformer.
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
extract_tags(html_content, tags)
Extract specific tags from a given HTML content.
remove_unnecessary_lines(content)
Clean up the content by removing unnecessary lines.
remove_unwanted_tags(html_content, unwanted_tags)
Remove unwanted tags from a given HTML content.
transform_documents(documents[, ...])
Transform a list of Document objects by cleaning their HTML content.
__init__() → None[source]¶
Initialize the transformer.
This checks if the BeautifulSoup4 package is installed.
If not, it raises an ImportError.
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
static extract_tags(html_content: str, tags: List[str]) → str[source]¶
Extract specific tags from a given HTML content.
Parameters
html_content – The original HTML content string.
tags – A list of tags to be extracted from the HTML.
Returns
A string combining the content of the extracted tags.
static remove_unnecessary_lines(content: str) → str[source]¶
static remove_unnecessary_lines(content: str) → str[source]¶
Clean up the content by removing unnecessary lines.
Parameters
content – A string, which may contain unnecessary lines or spaces.
Returns
A cleaned string with unnecessary lines removed.
static remove_unwanted_tags(html_content: str, unwanted_tags: List[str]) → str[source]¶
Remove unwanted tags from a given HTML content.
Parameters
html_content – The original HTML content string.
unwanted_tags – A list of tags to be removed from the HTML.
Returns
A cleaned HTML string with unwanted tags removed.
transform_documents(documents: Sequence[Document], unwanted_tags: List[str] = ['script', 'style'], tags_to_extract: List[str] = ['p', 'li', 'div', 'a'], remove_lines: bool = True, **kwargs: Any) → Sequence[Document][source]¶
Transform a list of Document objects by cleaning their HTML content.
Parameters
documents – A sequence of Document objects containing HTML content.
unwanted_tags – A list of tags to be removed from the HTML.
tags_to_extract – A list of tags whose content will be extracted.
remove_lines – If set to True, unnecessary lines will be
content. (removed from the HTML) – 
Returns
A sequence of Document objects with transformed content.
Examples using BeautifulSoupTransformer¶
Beautiful Soup
Set env var OPENAI_API_KEY or load from a .env file:
langchain_community.document_transformers.embeddings_redundant_filter.EmbeddingsClusteringFilter¶
class langchain_community.document_transformers.embeddings_redundant_filter.EmbeddingsClusteringFilter[source]¶
Bases: BaseDocumentTransformer, BaseModel
Perform K-means clustering on document vectors.
Returns an arbitrary number of documents closest to center.
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param embeddings: langchain_core.embeddings.Embeddings [Required]¶
Embeddings to use for embedding document contents.
param num_closest: int = 1¶
The number of closest vectors to return for each cluster center.
param num_clusters: int = 5¶
Number of clusters. Groups of documents with similar meaning.
param random_state: int = 42¶
Controls the random number generator used to initialize the cluster centroids.
If you set the random_state parameter to None, the KMeans algorithm will use a
random number generator that is seeded with the current time. This means
that the results of the KMeans algorithm will be different each time you
run it.
param remove_duplicates: bool = False¶
By default duplicated results are skipped and replaced by the next closest
vector in the cluster. If remove_duplicates is true no replacement will be done:
This could dramatically reduce results when there is a lot of overlap between
clusters.
param sorted: bool = False¶
By default results are re-ordered “grouping” them by cluster, if sorted is true
result will be ordered by the original position from the retriever
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document]¶
Asynchronously transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep – set to True to make a deep copy of the model
Returns
new model instance
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
classmethod from_orm(obj: Any) → Model¶
classmethod from_orm(obj: Any) → Model¶
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Filter down documents.
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using EmbeddingsClusteringFilter¶
langchain_community.document_transformers.doctran_text_extract.DoctranPropertyExtractor¶
class langchain_community.document_transformers.doctran_text_extract.DoctranPropertyExtractor(properties: List[dict], openai_api_key: Optional[str] = None, openai_api_model: Optional[str] = None)[source]¶
Extract properties from text documents using doctran.
Parameters
properties – A list of the properties to extract.
openai_api_key – OpenAI API key. Can also be specified via environment variable
OPENAI_API_KEY.
Example
from langchain_community.document_transformers import DoctranPropertyExtractor
properties = [
    {
        "name": "category",
        "description": "What type of email this is.",
        "type": "string",
        "enum": ["update", "action_item", "customer_feedback", "announcement", "other"],
        "required": True,
    },
    {
        "name": "mentions",
        "description": "A list of all people mentioned in this email.",
        "type": "array",
        "items": {
            "name": "full_name",
            "description": "The full name of the person mentioned.",
            "type": "string",
        },
        "required": True,
    },
    {
        "name": "eli5",
        "description": "Explain this email to me like I'm 5 years old.",
        "type": "string",
        "required": True,
    },
]
# Pass in openai_api_key or set env var OPENAI_API_KEY
property_extractor = DoctranPropertyExtractor(properties)
transformed_document = await qa_transformer.atransform_documents(documents)
Methods
__init__(properties[, openai_api_key, ...])
Methods
__init__(properties[, openai_api_key, ...])
atransform_documents(documents, **kwargs)
Extracts properties from text documents using doctran.
transform_documents(documents, **kwargs)
Transform a list of documents.
__init__(properties: List[dict], openai_api_key: Optional[str] = None, openai_api_model: Optional[str] = None) → None[source]¶
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Extracts properties from text documents using doctran.
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
Examples using DoctranPropertyExtractor¶
Doctran Extract Properties
langchain_community.document_transformers.nuclia_text_transform.NucliaTextTransformer¶
class langchain_community.document_transformers.nuclia_text_transform.NucliaTextTransformer(nua: NucliaUnderstandingAPI)[source]¶
The Nuclia Understanding API splits into paragraphs and sentences,
identifies entities, provides a summary of the text and generates
embeddings for all sentences.
Methods
__init__(nua)
atransform_documents(documents, **kwargs)
Asynchronously transform a list of documents.
transform_documents(documents, **kwargs)
Transform a list of documents.
__init__(nua: NucliaUnderstandingAPI)[source]¶
async atransform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Asynchronously transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
transform_documents(documents: Sequence[Document], **kwargs: Any) → Sequence[Document][source]¶
Transform a list of documents.
Parameters
documents – A sequence of Documents to be transformed.
Returns
A list of transformed Documents.
Examples using NucliaTextTransformer¶
Nuclia Understanding API document transformer
langchain_community.vectorstores.dingo.Dingo¶
class langchain_community.vectorstores.dingo.Dingo(embedding: Embeddings, text_key: str, *, client: Any = None, index_name: Optional[str] = None, dimension: int = 1024, host: Optional[List[str]] = None, user: str = 'root', password: str = '123123', self_id: bool = False)[source]¶
Dingo vector store.
To use, you should have the dingodb python package installed.
Example
from langchain_community.vectorstores import Dingo
from langchain_community.embeddings.openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
dingo = Dingo(embeddings, "text")
Initialize with Dingo client.
Attributes
embeddings
Access the query embedding object if available.
Methods
__init__(embedding, text_key, *[, client, ...])
Initialize with Dingo client.
aadd_documents(documents, **kwargs)
Run more documents through the embeddings and add to the vectorstore.
aadd_texts(texts[, metadatas])
Run more texts through the embeddings and add to the vectorstore.
add_documents(documents, **kwargs)
Run more documents through the embeddings and add to the vectorstore.
add_texts(texts[, metadatas, ids, text_key, ...])
Run more texts through the embeddings and add to the vectorstore.
adelete([ids])
Delete by vector ID or other criteria.
afrom_documents(documents, embedding, **kwargs)
Return VectorStore initialized from documents and embeddings.
afrom_texts(texts, embedding[, metadatas])
Return VectorStore initialized from texts and embeddings.
amax_marginal_relevance_search(query[, k, ...])
amax_marginal_relevance_search(query[, k, ...])
Return docs selected using the maximal marginal relevance.
amax_marginal_relevance_search_by_vector(...)
Return docs selected using the maximal marginal relevance.
as_retriever(**kwargs)
Return VectorStoreRetriever initialized from this VectorStore.
asearch(query, search_type, **kwargs)
Return docs most similar to query using specified search type.
asimilarity_search(query[, k])
Return docs most similar to query.
asimilarity_search_by_vector(embedding[, k])
Return docs most similar to embedding vector.
asimilarity_search_with_relevance_scores(query)
Return docs and relevance scores in the range [0, 1], asynchronously.
asimilarity_search_with_score(*args, **kwargs)
Run similarity search with distance asynchronously.
delete([ids])
Delete by vector IDs or filter.
from_documents(documents, embedding, **kwargs)
Return VectorStore initialized from documents and embeddings.
from_texts(texts, embedding[, metadatas, ...])
Construct Dingo wrapper from raw documents.
max_marginal_relevance_search(query[, k, ...])
Return docs selected using the maximal marginal relevance.
max_marginal_relevance_search_by_vector(...)
Return docs selected using the maximal marginal relevance.
search(query, search_type, **kwargs)
Return docs most similar to query using specified search type.
similarity_search(query[, k, search_params, ...])
Return Dingo documents most similar to query, along with scores.
similarity_search_by_vector(embedding[, k])
Return docs most similar to embedding vector.
similarity_search_with_relevance_scores(query)
Return docs and relevance scores in the range [0, 1].
Return docs and relevance scores in the range [0, 1].
similarity_search_with_score(query[, k, ...])
Return Dingo documents most similar to query, along with scores.
__init__(embedding: Embeddings, text_key: str, *, client: Any = None, index_name: Optional[str] = None, dimension: int = 1024, host: Optional[List[str]] = None, user: str = 'root', password: str = '123123', self_id: bool = False)[source]¶
Initialize with Dingo client.
async aadd_documents(documents: List[Document], **kwargs: Any) → List[str]¶
Run more documents through the embeddings and add to the vectorstore.
Parameters
(List[Document] (documents) – Documents to add to the vectorstore.
Returns
List of IDs of the added texts.
Return type
List[str]
async aadd_texts(texts: Iterable[str], metadatas: Optional[List[dict]] = None, **kwargs: Any) → List[str]¶
Run more texts through the embeddings and add to the vectorstore.
add_documents(documents: List[Document], **kwargs: Any) → List[str]¶
Run more documents through the embeddings and add to the vectorstore.
Parameters
(List[Document] (documents) – Documents to add to the vectorstore.
Returns
List of IDs of the added texts.
Return type
List[str]
add_texts(texts: Iterable[str], metadatas: Optional[List[dict]] = None, ids: Optional[List[str]] = None, text_key: str = 'text', batch_size: int = 500, **kwargs: Any) → List[str][source]¶
Run more texts through the embeddings and add to the vectorstore.
Parameters
texts – Iterable of strings to add to the vectorstore.